1. Technical Field
The present invention relates to a method for separating the diffuse reflection component and the specular reflection component in images obtained by shooting a subject through a polarizing filter by utilizing probabilistic independence between them, and more particularly relates to a technique enabling reflection component separation in a general illumination condition.
2. Background Art
Reflected light from an object can be expressed by a sum of two components of the specular reflection component and the diffuse reflection component. Reflection by the two components is proposed as a dichromatic reflection model in Non-patent Document 1. The specular reflection component is light reflected at a boundary between the surface of an object and an air layer while the diffuse reflection component is light which passes through the surface of the object, are made incident to the inside of the object, repeats reflection by pigments, and then returned to the air.
Almost all image processing algorithms, such as corresponding point search and photometric stereo premise diffuse reflection, but the specular reflection serves as a factor of lowering performance of various kinds of image processing algorithms. For this reason, various kinds of techniques for separating the specular reflection component and the diffuse reflection component have been proposed.
For separating the specular reflection component and the diffuse reflection component, two types of methods have been proposed: namely, (A) methods utilizing color characteristics of reflected light; and (B) methods utilizing polarization characteristics of specular reflection.
In the methods (A), the reflection components are separated by utilizing difference in color vector on the ground that in a case of a dielectric subject, the diffuse reflected light is reflected at an object color while the specular reflected light is reflected at an illumination color. The methods, however, cannot separate them well when the object color is similar to the illumination color or another object is reflected by inter-reflection. Further, when a subject is made of metal, the separation methods utilizing the color characteristics cannot be applied because some kinds of metal have so large wavelength dependencies of a Fresnel coefficient that the specular reflected light and the illumination light do not agree with each other in color vector.
On the other hand, in the methods (B), the reflection components are separated by utilizing the fact that the specular reflection component are polarized usually while the diffuse reflection component can be regarded as unpolarized light usually. Non-patent Document 2, Non-patent Document 3, Patent Document 1, Patent Document 2, and the like propose methods utilizing polarization. In these methods utilizing polarization, image processing independent from the colors of illumination and an object can be performed, which is the difference from the methods utilizing the color characteristics.
Non-patent Document 2 utilizes the fact that light reflected on the surface of a glass is polarized more strongly than light passing through the glass, and an image reflected in the glass and an image of an object therebehind are separated from each other by obtaining a minimum intensity of each pixel from a plurality of images shot with a polarizing filter mounted to a camera rotated. However, an image reflected in the glass can be separated well only when the incident angle to the glass approximates to a Brewster's angle.
In Non-patent Document 3, a minimum intensity and a maximum intensity are obtained in each pixel of a plurality of images shot with a polarizing filter mounted to a camera rotated, a ratio of a parallel component to a perpendicular component of a Fresnel coefficient is estimated on the basis of the ratio of the intensities so that the specular reflected light and the diffuse reflected light from a subject are separated even when the incident angle is one other than a Brewster's angle. This method stands on the assumption that the diffuse reflection component is uniform in a region where specular reflection is present and, therefore, cannot be applied to a subject having texture of diffuse reflection.
Patent Document 1 uses a polarizing filter for each of an illumination and a camera. A minimum intensity is obtained in each pixel of a plurality of images shot with the polarizing filter mounted to the camera rotated and with the illumination light polarized linearly, and an image composed of the obtained minimum intensities is recorded as a diffuse reflection image. This method utilizes a property of specular reflected light which is polarized completely irrespective of the incident angle when illumination light is polarized completely. This method, however, cannot remove specular reflected light from light other than completely polarized illumination light and, therefore, is utilized only in a mat and black light blocking box, which means inapplicability to a general condition.
In Patent Document 2, on the assumption that there is probabilistic independence between the specular reflection component and the diffuse reflection component, both the components are separated from a plurality of images shot through a polarizing filter. This method can separate the reflection components at an angle other than a Brewster's angle, which is different from Non-patent Document 2, and is applicable to a subject having texture of diffuse reflection, which is different from Non-patent Document 3, and requires no polarization of illumination light, which is different from Patent Document 1.
Prior to description of the method of Patent Document 2, general description will be given to reflection of light from an object and polarization.
In the previously mentioned dichromatic reflection model, the reflected light from an object is expressed by a sum of the two components of the diffuse reflection component and the specular reflection component. The diffuse reflection component Ldiff is expressed by a Lambertian model expressed by Expression 1 usually.Ldiff=κdiffmax[0,(Li cos θi)]  Expression 1
Wherein, Li is an intensity of incident light, θi is an incident angle, and κdiff is a diffuse reflectance of an object. The diffuse reflection component can be regarded as unpolarized light usually.
On the other hand, the specular reflection component Lspec is expressed by a Torrance-Sparrow model expressed by Expression 2 usually
                              L          spec                =                              cF            ⁡                          (                                                θ                  i                  ′                                ,                η                            )                                ⁢          G          ⁢                                          ⁢                                    L              i                                      cos              ⁢                                                          ⁢                              θ                r                                              ⁢                      exp            (                          -                                                α                  2                                                  2                  ⁢                                      σ                    α                    2                                                                        )                                              Expression        ⁢                                  ⁢        2            
Wherein, Li is an intensity of incident light as well as in the Lambertian model, θr is an angle formed between the normal line of the surface of an object and a view-line direction, a is an angle formed between the normal direction of a microfacet forming the specular reflection and the normal direction of the surface of an object, and σα is a parameter representing roughness of the surface of an object. Further, F is a Fresnel coefficient about reflected energy and depends on a relative refractive index η, which is a ratio of an incident angle θ′i at a microfacet to refractive indices η1 and η2 of two media. F is a real number when the media are dielectric, and is represented by n hereinafter. G is a geometric attenuation factor and expresses influence of incident light masking and reflected light shadowing by a microfacet. c is a constant representing a ratio of diffuse reflection and specular reflection.
The specular reflection component is polarized in the presence of difference in Fresnel coefficient between a component parallel to a light incident plane and a component perpendicular thereto. The Fresnel coefficients about reflected energy with respect to the parallel component and the perpendicular component are expressed by Expression 3 and Expression 4, respectively.
                              F          p                =                                            tan              2                        ⁡                          (                                                θ                  i                  ′                                -                                  θ                  i                  ″                                            )                                                          tan              2                        ⁡                          (                                                θ                  i                  ′                                +                                  θ                  i                  ″                                            )                                                          Expression        ⁢                                  ⁢        3                                          F          s                =                                            sin              2                        ⁡                          (                                                θ                  i                  ′                                -                                  θ                  i                  ″                                            )                                                          sin              2                        ⁡                          (                                                θ                  i                  ′                                +                                  θ                  i                  ″                                            )                                                          Expression        ⁢                                  ⁢        4            
Wherein, θ″i is an angle of refraction on a microfacet, and Expression 5 is held from Snell's law.
                                          sin            ⁢                                                  ⁢                          θ              i              ′                                            sin            ⁢                                                  ⁢                          θ              i              ″                                      =                                            n              2                                      n              1                                =          n                                    Expression        ⁢                                  ⁢        5            
Wherein n1 and n2 are a refractive index of a media 1 on the incident side and a refractive index of a media 2 on the refraction side. The ratio of the refractive indices n2 to n1 is represented hereinafter by a relative refractive index n.
FIG. 26 is a graph of Fresnel coefficient with respect to the parallel component and the perpendicular component at an incident plane when the refractive index n is 1.6.
From the foregoing, the intensity Lr of reflected light to incident light as unpolarized light can be expressed by Expression 6. Wherein, L′spec is obtained by subtracting the term of Fresnel intensity reflectance from the specular reflection component Lspec expressed by Expression 2.
                              L          r                =                              L            diff                    +                                    1              2                        ⁢                          (                                                                    F                    p                                    ⁡                                      (                                                                  θ                        i                        ′                                            ,                      n                                        )                                                  +                                                      F                    s                                    ⁡                                      (                                                                  θ                        i                        ′                                            ,                      n                                        )                                                              )                        ⁢                          L              spec              ′                                                          Expression        ⁢                                  ⁢        6            
In the case where reflected light from an object is observed through a polarizing filter, the intensity thereof is an intensity of a component parallel to the principal axis of the polarizing filter in reflected light before passing through the polarizing filter. When the reflected light from the object is observed through the rotating polarizing filter, the intensity varies as in a sine function as in FIG. 27 according to the angle of the principal axis of the polarizing filter. As shown in FIG. 26, the perpendicular component of the reflected light is larger than the parallel component thereof with respect to an incident plane at almost all incident angles. Accordingly, the intensity of the observed specular reflection component is minimum when the principal axis of the polarizing filter is parallel to the incident plane (when an angle ψ formed between the principal axis of the polarizing filter and the incident plane is 0 rad) while being maximum when it is perpendicular thereto (when the angle ψ formed between the principal axis of the polarizing filter and the incident plane is π/2 rad). The diffuse reflection component can be regarded as unpolarized light and, accordingly, is constant regardless of the direction of the principal axis of the polarizing filter.
In a Torrance-Sparrow model, it is supposed that specular reflection is caused by reflecting incident light at a microfacet on the surface of an object. Accordingly, the incident plane and the local incident angle of the specular reflection component in each pixel of an observed image are determined by the direction of a light source and the view-line direction of a camera and is independent from the normal direction of the surface of an object.
The method of Patent Document 2 assumes a light source being sufficiently far away and an orthographic model being as a camera model. In this case, as shown in FIG. 33, the incident plane and the local incident angle can be considered to be the same in every pixel in which specular reflection is present in a shot image. Herein, the words “the incident plane is the same” mean that directions in which the incident planes of target pixels are projected onto an image plane as in FIG. 5 are equal to each other in every pixel in which specular reflection is present. At the same time, it is assumed that the refractive index of an object is uniform. Under the above assumptions, the Fresnel coefficients of all the pixel corresponding to an object become the same, and every pixel has the same phase of intensity variation of specular reflection when the polarizing filter is rotated.
Herein, when Id, Is, and I are a diffuse reflection image vector, a specular reflection image vector, and an observed image vector, respectively, the relationship expressed by Expression 7 is held. Each vector Id, Is, I is a column vector having dimensions of which number is equal to the number of pixels. f(ψ) is a function (scalar) of an angle ψ formed between the principal axis of a polarizing filter and an incident plane.I=Id+ƒ(ψ)Is  Expression 7
FIG. 28 shows the relationship thereamong. The observed image vector increases and decreases in the range between f(0)Is and f(π/2)Is. When a plurality of image vectors, for example, two image vectors I1, I2 are observed through a polarizing filter having principal axes different from each other, the direction of the specular reflection image vector Is is obtained from the difference (I1−I2) therebetween, as shown in FIG. 29. Accordingly, when the vector of this difference is multiplied by an appropriate constant k and is subtracted from the observed image vector, the diffuse reflection image vector Id can be obtained.
The method of Patent Document 2 obtains, on the assumption of the probabilistic independence between the diffuse reflection image vector Id and the specular reflection image vector Is, the coefficient f(ψ) and both the vectors. Specifically:
(1) A matrix having row vectors of a plurality of observed image as elements is decomposed into the form of a product by utilizing a maximum rank decomposition.
(2) An arbitrary 2×2 nonsingular matrix is introduced to determine temporally a diffuse reflection image vector candidate and a specular reflection image vector candidate.
(3) By evaluating mutual information content of them, a diffuse reflection image vector, a specular reflection image vector, and a coefficient f(ψ) matrix when the mutual information content is minimum (probabilistic independence is maximum) are employed as estimation values.Patent Document 1: Japanese Patent Application Laid Open Publication No. 11-41514APatent Document 2: Japanese Patent No. 3459981Non-patent Document 1: S. Shafer, “Using color to separate reflection components,” Color Research and Applications, vol. 10, no. 4, pp. 210-218, 1985Non-patent Document 2: Masaki Iwase, Tsuyoshi Yamamura, Toshimitsu Tanaka, and Noboru Ohnishi, “A camera system for separating real and specularly reflected images from their mixing images,” the transaction of the Institute of Electronics, Information and Communication, D-II, Vol. J81-D-II, No. 6, pp. 1224-1232, June, 1998Non-patent Document 3: L. B. Wolff, “Using polarization to separate reflection components,” Proc. IEEE conf. CVPR, pp. 363-369, 1989